Moving picture coding apparatus and moving picture decoding apparatus

ABSTRACT

A video encoder ( 70 ) for coding moving pictures comprising a buffer ( 16   c ) with a plurality of memory areas capable of storing frames composed of top fields and bottom fields, a motion estimation unit ( 19 ) operable to code, field by field, inputted pictures performing moving estimation and moving compensation by referring, field by field, to the picture data stored in a memory area, a motion compensation unit ( 16   d ), a subtractor ( 11 ), a transformation unit ( 13 ) and a quantization unit ( 14 ), a memory management unit ( 71 ) operable to manage, frame by frame, a plurality of memory areas, an inverse quantization unit ( 16   a ) and inverse discrete cosine transform unit ( 16   b ) operable to decode picture data in coded fields and store the picture data in the decoded field in any of the plurality of memory areas under the management by the memory management unit ( 71 ).

TECHNICAL FIELD

The present invention relates to memory management control of a multiframe buffer used to store reference picture data in video coding anddecoding, and particularly to a memory management method for interlacedvideo data.

BACKGROUND ART

Moving pictures are used in an increasing number of applications infields ranging from video telephony and video conferencing to DVD anddigital television. In order to transmit moving pictures, it isnecessary to transmit an enormous volume of data via an existingtransmission channel in which an effective frequency bandwidth islimited. When digital data is transmitted within a limited transmissionband, it becomes an absolute necessity to compress or reduce the volumeof data to be transmitted.

For the purpose of enabling inter-operability between video data in aplurality of systems dedicated to applications which were designed bydifferent manufacturers, video coding standards have been developed forcompressing the volume of video data in a common method. Such videocoding standards include H.261 and H.263 developed by ITU, and MPEG-1,MPEG-2, and MPEG-4 developed by ISO/IEC.

A basic approach to coding taken by many of the above standards iscomprised of the following major stages:

1. dividing each picture into blocks made up of pixels so as to enableprocessing to be performed at a block level on pictures that constitutevideo. A picture refers to a frame or fields;

2. reducing spatial redundancy in a picture by performingtransformation, quantization, and entropy coding on video data in oneblock; and

3. Coding a difference between consecutive frames, utilizing correlationbetween the consecutive pictures.

The above is achieved by the use of a motion estimation and compensationtechnique. In order to estimate, on a block-by-block basis, a motionvector indicating predictive image data which indicates a strongcorrelation between frames, an encoder performs motion estimation tosearch the coded frames for a position of image data indicating a strongcorrelation. Furthermore, the encoder and the decoder perform motioncompensation to extract predictive image data with respect to the motionvector.

FIG. 1 shows an example configuration of a video encoder (moving picturecoding apparatus). The video encoder illustrated in the diagram iscomprised of: a transform unit 13 operable to transform spatial videodata into a frequency domain; a quantization unit 14 operable toquantize the transform coefficients obtained by the transform unit 13; avariable length coding unit 15 operable to perform entropy coding on thequantized transform coefficients; a video buffer 17 for supplying atransmission channel with compressed video data at a variable bit ratedepending on a transmission rate; a decoder 16, and a motion estimationunit 19.

Video data 10 from the encoder shown in FIG. 1 is inputted in a form ofpixel values using pulse-code modulation (PCM). A subtractor 11calculates a differential value between the video data 10 and amotion-compensated image 12. The motion-compensated image 12 is obtainedas a result of decoding an already coded image and performs motioncompensation on the resultant (“a current decoded picture”). This iscarried out by a decoder 16 in pairs with the video encoder. The decoder16 performs the coding procedure in an inverse order. Stated anotherway, the decoder 16 is comprised of an inverse quantization unit (Q-1),an inverse discrete cosine transform unit (IDCT), and an adder foradding a decoded difference and a motion-compensated image so as togenerate a preceding picture which is equivalent to the one obtained atthe decoder's side.

In motion-compensated coding, motion-compensated data in a currentpicture is generated, based on motion estimation which has beenperformed on such picture and a decoded picture, from picture dataderived from the corresponding decoded picture. A motion predicted valueis represented by a two-dimensional motion vector indicating a pixeldisplacement between the decoded picture and the current picture.Usually, motion estimation is performed on a block-by-block basis.Stated another way, a block in the decoded picture which is moststrongly correlated with a block in the current frame is regarded as amotion-compensated image. The motion estimation unit 19 operable toperform such motion estimation and a motion compensation unit MCoperable to generate a motion-compensated image from the picture whichhas been decoded corresponding to the motion vector are incorporatedinto the encoder.

The video encoder illustrated in FIG. 1 operates in the followingmanner. The video image of the video signal 10 is divided into a groupof a certain number of small blocks generally called macro block. Forexample, a video image 20 shown in FIG. 2 is divided into a plurality ofmacro blocks 21. Generally, each of the macro blocks has a size of 16×16pixels.

Furthermore, a picture is divided into a certain number of slices 22.Each slice is made up of a plurality of macro blocks and serves as aunit of alignment recovery at the time of data loss. Note that anarrangement of macro blocks that constitute a slice is not necessarilymade up of macro blocks in the same row as shown in FIG. 2, andtherefore that it is also possible that a slice includes macro blocks ina plurality of rows and there is a delimiter of another slice in themiddle of the row.

When image data in video is coded by just reducing the volume of spatialredundancy in the image, the resultant picture is called I picture. An Ipicture is coded with reference to only pixel values in the picture. Thedata size of a coded I picture is large because temporal informationused to reduce the volume of data cannot be used for an I picture.

With the aim of performing an efficient compression utilizing temporalredundancy between consecutive pictures, prediction coding is performedon the consecutive pictures on the basis of motion estimation and motioncompensation. When a reference picture selected in motion estimation isone picture which has been already coded and decoded, it is called Ppicture. Meanwhile, when two pictures are reference pictures (usually,forward and backward pictures in display order with respect to a currentpicture), they are called B picture.

According to the H.26L standard on a picture coding method underdevelopment, motion compensation for each of 16×16 macro blocks can becarried out by using a different block size. Each motion vector can bedetermined with respect to a block with a size of 4×4, 4×8, 8×4, 8×8,8×16, or 16×16 pixels. The effect of using a smaller block size formotion compensation is that it becomes possible to describe detailedmotions.

Based on the result of motion estimation, estimation is performed on adetermined motion vector as motion compensation. Subsequently,information included in a prediction error block obtained from thepredicted block is transformed into transform coefficients in thetransformation unit 13. Generally, two-dimensional Discrete CosineTransform (DCT) is employed. Such obtained transform coefficients arequantized, and entropy coding (VLC) is performed on the resultant by theentropy coding unit 15 in the end. Note that a motion vector calculatedby the motion estimation unit 19 is used for motion compensation and isincorporated into compression video data 18 via the variable lengthcoding unit 15 and the video buffer 17.

A transmission stream of the compressed video data 18 is transmitted tothe decoder (picture decoding apparatus), where a sequence of codedvideo images is reproduced on the basis of the received data. Theconfiguration of the decoder pairs with that of the decoder 16 includedin the video encoder shown in FIG. 1.

In a new video coding method, it is possible to use a plurality ofbi-directionally predictive pictures so as to realize more efficientpicture coding. For this reason, a motion estimation unit and a motioncompensation unit include multi frame buffers for providing a variety ofreference pictures. Information indicating individual reference image isadded to a motion vector.

The internal structure of a multi frame buffer is as shown by FIG. 3,and the figure with a reference number 30 shows the whole structure. Themulti frame buffer is composed of a plurality of memory areas 31 and 32for storing frames of the video signal. The memory areas in the multiframe buffer 30 are divided into two different kinds of memory areas,that is, a short term picture memory area 33 mainly for storingreference pictures used as a reference picture for a short term and along term picture memory area 34 mainly for storing reference picturesused as a reference picture for a long term.

The multi frame buffer stores reference pictures selected as appropriateso as to code or decode special pictures. The procedure for storingreference pictures is divided into two processing stages, that is, (1) astage of realigning reference pictures and (2) a stage of bufferingreference pictures.

(1) the reference pictures are aligned based on the reference pictureorder information to be transmitted in the slice layer. Orderingreference pictures has influence on the coding or decoding processing ofa group of macro blocks included in one slice. The aim of thisprocessing is to reduce the number of bits of information that indicatesa reference picture to be referred to at the time of motion compensationby assigning a smaller number to a picture to be frequently referred to,in other words, by assigning a reference number with a shorter signallength to a picture with a smaller number.

(2) As for buffering reference pictures, buffering pictures to be codedor decoded is controlled when updating reference pictures stored in themulti frame buffer for each coding or decoding processing.

With the aim of buffering reference pictures, one of the two differentkinds of memory management control mode, that is, “a shift windowbuffering mode” or “adaptive memory control buffering mode” can be used.

In the shift window buffering mode, the pictures as targets of eachcoding or decoding are stored in the multi frame buffer. The picture inthe short term picture memory area of the multi frame buffer isperiodically replaced by a new picture in a First-In First-Out (FIFO)method. There is no need to delete any picture data so as to storepictures under processing as long as the buffer has sufficient capacityof an unused memory area. If the unused area of the multi frame bufferbecomes full with new picture data that have already been processed, thestored picture data are being replaced by picture data of new picturesunder coding or decoding in the order of storage.

In the adaptive memory control buffering mode, they are stored in themulti frame buffer, or each picture to be deleted from the place isexplicitly selected. Memory control is performed according to the memorymanagement control processing parameter that enables memory managementcontrol in the coding side and the decoding side that are correlatedwith each other. In order to perform replacement processing of suchpictures, a unique identification number for explicitly specifying apicture to be coded or decoded is assigned to each memory area. Notethat an index indicating the picture order after realigning referencepictures of the above-mentioned (1) is assigned to each memory area, andthe index is called reference index.

Several problems are left in the above memory management control mode.In the conventional memory management control mode, it is impossible toefficiently process especially interlace video data. The interlace videodata comprises frames composed of two fields (a top field and a bottomfield), each of which has different time and a different verticalspatial location. Pictures may be coded field by field, which leads to aproblem of making memory management complicated.

DISCLOSURE OF INVENTION

Therefore, an object of the present invention is to provide a movingpicture coding apparatus, a moving picture decoding apparatus, a memorymanagement device and the like that make it possible to efficientlyperform picture memory management in coding and decoding on interlacevideo data.

In order to achieve the above-mentioned object, the moving picturecoding apparatus concerning the present invention comprises: a memoryunit with a plurality of memory areas capable of storing frames composedof top fields and bottom fields; a coding unit operable to code, fieldby field, inputted pictures, with performing motion estimation andmotion compensation by referring, field by field, to the picture datastored in the memory areas; a memory management unit operable to manage,frame by frame, the plurality of memory areas; and a storage unitoperable to decode field picture data coded by the coding unit and storethe decoded field picture data in any of the plurality of memory areasunder management of the memory management unit.

Also, the moving picture decoding apparatus concerning the presentinvention comprises: a memory unit with a plurality of memory areascapable of storing frames composed of top fields and bottom fields; adecoding unit operable to decode, field by field, inputted pictures,with performing motion compensation by referring, field by field, to thepicture data stored in the memory areas; a memory management unitoperable to manage, frame by frame, the plurality of the memory areas;and a storage unit operable to store the picture data in the fielddecoded by the decoding unit in any of the plurality of the memory areasunder the management of the memory management unit.

Further, the memory management device concerning the present inventionis a memory management device in the moving picture coding or decodingapparatus, comprising: a memory unit with a plurality of memory areascapable of storing frames composed of top fields and bottom fields, amanagement unit operable to manage, frame by frame, the plurality of thememory areas, and a memory access unit operable to store/read out fieldsin/from any of the plurality of the memory areas under management of themanagement unit.

These moving picture coding apparatus, moving picture decoding apparatusand a memory management device always perform memory management frame byframe irrespective of which unit of frame and field is used when storingpicture data to each memory area capable of storing one frame data ortwo field data (a top field data and a bottom field data) and readingout the picture data from each of the memory areas, and thus memorymanagement in coding and decoding interlace video data becomes simplerand more efficient. Especially, when both pictures coded frame by frameand pictures coded field by field are included in the interlace videodata, efficient memory management is realized because only memorycontrol on the basis of a frame unified as the unit is required.

Here, the memory management unit manages the picture data in each of theabove-mentioned plurality of the memory areas frame by frame so as tograsp whether they can be stored or not, and the storage unit may storethe field in the memory areas specified as storable by the memorymanagement unit. In other words, the memory management unit manages thememory areas by setting the storaged flag associated with each of theabove-mentioned plurality of the memory areas as “used” or “unused”, andthe storage unit may store the field in the memory areas specified as“unused” by the storaged flag. For example, the memory management unitsets the storaged flag associated with the memory area as “used” whenstoring the frame, a top field or a bottom field in the memory area, andsets the storaged flag associated with the memory are as “unused” whenno picture data is stored in the memory area, the stored frame becomesuseless, or both of a top field and a bottom field become useless. Bydoing so, 1 bit of storaged flag is assigned to the memory area thatstores two field data, only judging or changing the condition of the 1bit storaged flag enables the management of whether picture data isstorable in each memory area or not, which makes it possible to simplifythe processing concerning the management and reduce the required memorycapacity for management.

Also, the memory management unit may manage whether each of theplurality of the memory areas is “short term picture memory area” forshort term reference or “long term picture memory area” for long termreference frame by frame. In other words, the memory management unit maymanage the memory area by setting long term flags associated with eachof the plurality of the memory areas as “short term picture memory area”or “long term picture memory area”. For example, the memory managementunit sets the long term flag as “long term picture memory area” whenchanging frames, top fields or bottom fields stored in the memory areafor which the long term flag is set as “short term area”. By doing so, 1bit of long term flag is assigned to the memory area that stores twofield data, only judging or changing the condition of the 1 bit longterm flag enables the management of the reference attributes (“for shortterm reference” or “for long term reference”) of each memory area, whichmakes it possible to simplify the processing concerning the managementand reduce the required memory capacity for management. Further, onlychanging the long term flag from “short term picture memory area” to“long term picture memory area” enables the picture data stored in theshort term picture memory area to be transmitted to the long termpicture memory area.

Note that the present invention can also be realized as a moving picturecoding method, a moving picture decoding method, a memory managementmethod, a program, a computer-readable storage medium where the programis stored besides the moving picture coding apparatus, moving picturedecoding apparatus and memory management device mentioned earlier.

FURTHER INFORMATION ABOUT TECHNICAL BACKGROUND TO THIS APPLICATION

European Patent application No. 02015606. 3, field Jul. 15, 2002 isincorporated herein by reference.

BRIEF DESCRIPTION OF DRAWINGS

These and other objects, advantages and features of the invention willbecome apparent from the following description thereof taken inconjunction with the accompanying drawings that illustrate a specificembodiment of the invention. In the Drawings:

FIG. 1 is a block diagram of a DPCM video encoder on which motioncompensation is performed.

FIG. 2 shows a subdivision of a video picture for coding and decodingprocessing.

FIG. 3 shows the whole structure of a multi frame buffer used for motionestimation and motion compensation.

FIG. 4 shows data stored in association with one memory area.

FIG. 5 shows a block diagram of a motion estimation and/or motioncompensation unit.

FIG. 6 shows an example of control method of a multi frame buffer.

FIG. 7 is a block diagram showing the structure of a video encoder inthe second embodiment.

FIG. 8A is a diagram which shows an example of video signals to be codedframe by frame;

FIG. 8B is a diagram which shows an example of video signals to be codedfield by field; and

FIG. 8C is a diagram which shows an example of a video signal to becoded in a state where both frames and fields are included.

FIG. 9 is a flow chart showing the operational procedure of a memorymanagement unit.

FIG. 10A is a table which shows an example of memory management inassociation with FIG. 8A;

FIG. 10B is a table which shows an example of memory management inassociation with FIG. 8B; and

FIG. 10C is a table which shows an example of memory management inassociation with FIG. 8C.

FIG. 11A is a diagram which shows the storage condition of a buffer whena video signal to be inputted is coded frame by frame;

FIG. 11B is a diagram which shows the storage condition of a buffer whena video signal to be inputted is coded field by field; and

FIG. 11C is a diagram which shows the storage condition of a buffer whena video signal to be inputted is coded in a state where both frames andfields are included.

FIG. 12 is a block diagram showing the structure of the video decoderconcerning the present invention.

FIG. 13 is an illustration concerning the storage medium for storing aprogram for realizing the video encoder and the video decoder concerningthe present invention in a computer system.

FIG. 14 is a block diagram showing the whole structure of a contentsupply system that realizes a content distribution service concerningthe present invention.

FIG. 15 is a diagram showing an example of a cellular phone concerningthe present invention.

FIG. 16 is a block diagram showing the structure of the cellular phone.

FIG. 17 is a diagram showing the configuration of the system for digitalbroadcasting concerning the present invention.

BEST MODE FOR CARRYING OUT THE INVENTION First Embodiment

Explanation on a video encoder (a moving picture coding apparatus) inthe first embodiment of the present invention will be made below withreference to figures. Note that the video encoder in the presentembodiment has a configuration shown in FIG. 1 and a characteristicmemory management function shown in the later-explained FIG. 5.Different points from the conventional video encoder will be focused onin the explanation below.

As shown above, FIG. 3 shows the whole structure of a multi framebuffer. The multi frame buffer 30 has a plurality of memory areas 31 and32 for storing reference pictures composed of two fields, that is, a topfield and a bottom field. The data stored respectively in the memoryareas 31 and 32 is as shown in FIG. 4. Each memory area 41 is used forstoring a top field 42 and a bottom field 43. Each field data isindependently written in or read out from the memory area.

A memory management unit 53 shown in FIG. 5 generates reference picturesappropriate for motion estimation and motion compensation. Whenfinishing coding and decoding each picture, a memory unit (multi framebuffer) 52 is updated according to the memory control information. Thememory area used for storing new pictures such as picture data to becoded or decoded is marked by setting a storaged flag 44 assigned toeach memory area one by one. The reference pictures necessary for motionestimation or motion compensation of a series of pictures are held in aframe buffer by canceling these storaged flags. When deleting picturesfrom the memory unit (multi frame buffer) 52, the storaged flag 44 isset as “unused”. The storaged flag 44 turns into a use state when eitherone of a field of a frame, both fields and a frame is set. The storagedflag 44 shows the state of a frame, what is stored there is managedmemory (field) by memory. When only a kind of field, that is, a topfield or a bottom field is stored there, it is possible to show that theother kind of field can be stored.

The storaged flag can be stored in the memory unit (multi frame buffer)52 or, instead of it, the memory management unit 53 associated with amemory area corresponding to a multi frame. Associating a storaged flag44 with a pair of a top field and a bottom field eliminates the additionof any hardware for processing interlace video data.

Further, a long term flag 45 may be assigned to each memory area 41.This flag 45 is stored in the memory management unit 53 associated withthe memory unit (multi frame buffer) 52 or each memory area. Using thislong term flag 45 shows that each memory area is stored in a long termreference picture. In this way, setting a long term flag 45 realizes ashifting processing of pictures from the short term picture memory area33 to the long term picture memory area 34.

Note that the long term picture memory area 34 stores a top field or abottom field in either a top and bottom field locations 42 or 43 in theembodiment of the present invention, which realizes efficientimprovement in storing long term picture data. This is achieved bysetting two storaged flags for each long term picture memory area thatmakes it possible to show that each field location is “unused”.

The example method of the memory management control is shown in FIG. 6.When finishing coding or decoding pictures in the step 61, as shown inthe step 62, memory control processing for a multi frame buffer isstarted. In the step 63, a memory area which stores two fieldsunnecessary for later field processing is selected first, next to this,the storaged flag 44 is set as “unused” in the step 64. When there is amemory area storing a kind of field data that composes the same frameand the other field area in the memory area is “unused”, the other fielddata may be stored in the unused field area. For example, when top fielddata of the same frame has already been stored in the memory area andthe bottom field data has not been stored yet, the bottom field may bestored in the unused bottom field area of the memory area.

In this way, it becomes possible to write in new picture data in eachmemory space of a pair of unused fields. This selection is made based onthe memory control information sent from the video encoder. The videoencoder can generate control information based on the definition in theemployed coding standard.

Further, the memory control information selects a memory area that haslong term picture data (step 65). After that, long term flags 45 inthese memory areas are set in the step 66.

In the step 67, fields to be processed are stored in each “unused” fieldlocation of the memory area according to a memory control parameter,after that, coding or decoding processing is resumed in the step 68.

As shown above, the present invention relates to motion estimation andmotion compensation unit using a plurality of reference pictures, andthe plurality of reference pictures are stored in the memory unit with amemory area of each reference picture. Each memory area stores videodata of a frame including a top field and a bottom field. Storage ofvideo data to the memory unit is controlled by the memory managementunit. A storaged flag that permits storing video data of the twoconsecutive fields in a memory area is assigned to each memory area. Bydoing so, coding, decoding, motion estimation and motion compensation ofpictures are performed field by field or frame by frame when the videosignal is interlace video data, but management of the memory unit(management of “used” or “unused” and the like) is performed frame byframe, which simplifies the processing of memory management.

Note that the storaged flag 44 and the long term flag 45 are explainedat the same time in the first embodiment, it is also possible to useonly one of those flags or employ another method.

Second Embodiment

Next, a video encoder and a video decoder in the second embodiment ofthe present invention will be explained. The present embodiment isdetailed version of the first embodiment, and it is the same as thefirst embodiment in that memory management is performed frame by frame.

FIG. 7 is a block diagram showing the structure of a video encoder 70 inthe second embodiment. The feature of this video encoder 70 is thatmemory management to a multi frame buffer is performed frame by frameeven when the structure of the inputted video signal 10 and the unit forcoding is frame by frame, field by field or a mixture of those. Thevideo encoder 70 comprises a subtractor 11, a transform unit 13, aquantization unit 14, a variable length coding unit 15, a video buffer17, a decoder 16, and a motion estimation unit 19 and a memorymanagement unit 71. The decoder 16 comprises an inverse quantizationunit 16 a, an inverse discrete cosine transform unit 16 b, a buffer 16 cand a motion compensation unit 16 d. Note that the same number isassigned to the same component as the one of the conventional videoencoder shown in FIG. 1 and the video encoder in the first embodiment,and the explanations on these components are omitted.

The video encoder 70 of this embodiment has a characteristic memorymanagement unit 71. This memory management unit 71 orders the motioncompensation unit 16 d and the motion estimation unit 19 to select thesame unit for processing as the unit based on the external informationshowing which unit (frame by frame, field by field or a mixture of theseunits) is used when coding the pictures of the inputted video signal 10and manages reference pictures stored in a multi frame buffer, that is,the buffer 16 c frame by frame. For example, as shown in FIG. 8A, whenpictures of the inputted video signal 10 are coded frame by frame in GOP(Group Of Picture or sequence), it controls the motion compensation unit16 d and the motion estimation unit 19 to process pictures frame byframe and performs memory management frame by frame. On the other hand,as shown in FIG. 8B, when the pictures of the inputted video signal 10are coded field by field in GOP (or sequence) or the like, it controlsthe motion compensation unit 16 d and the motion estimation unit 19 toprocess pictures field by field and performs memory management frame byframe. Further, as shown in FIG. 8C, when the pictures of the inputtedvideo signal 10 are coded under the condition where both frames andfields are included for every GOP (or sequence), it controls the motioncompensation unit 16 d and the motion estimation unit 19 to processpictures frame by frame or field by field according to the unit operableto code in this case and performs memory management frame by frame.

Note that the details of the memory management are basically the same asthe first embodiment. Here, one bit of storaged flag and one bit of longterm flag are assigned to every memory area that is a unit operable tostore one frame data or two field data (top field data and bottom fielddata), the condition of “used” or “unused” and the storage area of “longterm” or “short term” are judged and recorded for every picture data(every frame or every pair of a top field and a bottom field in thisembodiment). These flags are set in the buffer 16 c or the memorymanagement unit 71. Note that “unused” shows that it is possible tostore picture data (frame data, top field data or bottom field data) inthe corresponding memory area, and that “used” shows that it isprohibited that the picture data be stored. Also, “long term picturememory area” and “short term picture memory area” corresponds to eacharea in the case of dividing the buffer 16 c into two kinds of storageareas, that is, a storage area mainly for long term reference and astorage area mainly for short term reference respectively.

FIG. 9 is a flow chart showing the operational procedure of the memorymanagement unit 71. The memory management unit 71 judges the unitoperable to code the video signal 10 to be inputted based on theinputted external information (step S70). When the pictures of the videosignal 10 are coded frame by frame (“frame by frame” in the step S70),the memory management unit 71 controls the motion compensation unit 16 dto perform motion compensation frame by frame and the motion estimationunit 19 to perform motion estimation frame by frame, and it stores oreliminates the reference pictures in or from the buffer 16 c frame byframe and stores them in the long term picture memory area or the shortterm picture memory area (step S71). On the other hand, when thepictures of the video signal 10 are coded field by field (“field byfield” in the step S70), the memory management unit 71 controls themotion compensation unit 16 d to perform motion compensation field byfield and the motion estimation unit 19 to perform motion estimationfield by field, it stores or eliminates the reference pictures in orfrom the buffer 16 c frame by frame and stores them in the long termpicture memory area or the short term picture memory area (step S72)

FIGS. 10A-10C are tables showing an example of memory management by thememory management unit 71. When the inputted video signal 10 is codedframe by frame, using the storaged flag and long term flag correspondingto every frame (that is, memory area) enables the memory management unit71 to record, refer to or update the state of “used” or “unused” as toevery frame and which area of “long term picture memory area” or “shortterm picture memory area” is used for storage as shown in FIG. 10A.

On the other hand, when the inputted video signal 10 is coded field byfield, using the storaged flag and long term flag or the likecorresponding to every pair of a top field and a bottom field (that is,memory area) enables the memory management unit 71 to record, refer toor update the state of “used” or “unused” as to every frame and whicharea of “long term picture memory area” or “short term picture memoryarea” is used for storage as shown in FIG. 10B. This detailed procedureis as shown in the flow chart of FIG. 6 in the first embodiment.

Further, when the inputted video signal 10 is coded using both frame andfield as the unit, using a storaged flag and a long term flag or thelike corresponding to every frame as to frames and a storaged flag and along term flag or the like corresponding to every pair of a top fieldand a bottom field as to fields enables the memory management unit 71 torecord, refer to or update the state of “used” or “unused” as to everyframe and which area of “long term picture memory area” or “short termpicture memory area” is used for storage for every frame or for everypair of a top field and a bottom field, that is, frame by frame as shownin FIG. 10C.

Here, specific control procedure of storaged flags are as follows: whenpicture data has not been stored in the memory area yet, when frame datastored in the memory area become unnecessary (when they are determinedto be not used), or when both of top field data and bottom field datastored in the memory area become unnecessary (both of field data aredetermined to be not used), the storaged flags corresponding to thememory area are set as “unused”. Doing so enables storing new frame dataor field data in the memory area.

On the other hand, when storing new frame data, top field data or bottomfield data in the “unused” memory area, the storaged flag correspondingto the memory area is set as “used”. By doing so, it is prohibited thatother frame data or field data be stored in the memory area, and it ismade certain that frame data or field data that have already been storedare to be held in the memory area. When top field data or bottom fielddata are stored in the “unused” memory area, the storaged flagcorresponding to the memory area is set as “used”, but it manages dataof the bottom field and top field that composes the same frame regardingthem as storable respectively in either one of the field areas or theother field area of the memory area.

Also, specific control procedure of long term flags are as follows: whenusing picture data stored in the memory area for short term reference,the short term flags corresponding to the memory area are set as the“short term picture memory area”, the picture data being the frame datawhen frame data are stored, the top field data when only top field dataare stored, the bottom field data when only bottom field data arestored, or both the top field data and the bottom field data when topfield data and bottom field data are stored. By doing so, it becomespossible to use the memory area as a buffer memory of FIFO along withthe memory area of other “short term picture memory area”.

On the other hand, when using picture data stored in the memory area forlong term reference, the long term flags corresponding to the memoryarea are set as the “long term picture memory area”, the picture databeing the frame data when frame data are stored, the top field data whenonly top field data are stored, the bottom field data when only bottomfield data are stored, or both the top field data and the bottom fielddata when top field data and bottom field data are stored. By doing so,it is made sure that the picture data are stored in the memory areauntil they are explicitly eliminated from the memory area (become“unused”). Also, when changing the frames, top fields or bottom fieldsthat are stored in the memory area where long term flags are set as“short term picture memory area” into flags for long term reference, thelong term flag should be set as the “long term picture memory area”.Doing so makes it possible to easily shift the picture data from the“short term picture memory area” to the “long term picture memory area”.

FIGS. 11A-11C are diagrams showing the storage condition of thereference pictures in a buffer. FIG. 11A shows the storage condition ofthe buffer 16 c in the case where the video signal 10 to be inputted iscoded frame by frame, FIG. 11B shows the storage condition of the buffer16 c in the case where the video signal 10 to be inputted is coded fieldby field, and FIG. 11C shows the storage condition of the buffer 16 c inthe case where the video signal 10 to be inputted is coded under thecondition that both frames and fields are included.

As shown above, the video encoder 70 in the embodiment performs memorymanagement frame by frame in any case where the unit operable to codethe inputted video signal 10 is frame by frame, field by field, or amixture of those. Therefore, compared to a complicated memory managementmethod under the condition that both a frame and a field are used as theunit, a processing load for memory management is reduced and a circuitsize for it is downsized.

The memory management unit 71 in the embodiment can be applied not onlyfor a video encoder but also for a video decoder (moving picturedecoding apparatus). FIG. 12 is a block diagram showing the structure ofthe video decoder 100 with the memory management unit 109 with the samefunction as the above-mentioned memory management unit 71. The videodecoder 100 comprises an input buffer 102 for saving the inputted codedvideo signal 101, a variable length decoding device 103 for decodingcorresponding to the coding by the variable length coding unit 15equipped in the video encoder 70, an inverse quantization unit 104operable to perform inverse quantization corresponding to thequantization by the quantization unit 14 equipped in the video encoder70, an inverse discrete cosine transform unit 105 operable to performinverse transform to the transformation performed by the transformationunit 13 equipped in the video encoder 70, an adder 106 for outputtingthe video signal 110 by adding pictures, a buffer 107 as a multi framebuffer for storing reference pictures, a motion compensation unit 108operable to perform motion compensation on the reference pictures storedin the buffer 107 using the motion vectors, which are contained in thecoding video signal 101, obtained through the variable length decodingdevice 103, and the memory management unit 109.

The memory management unit 109 orders the motion compensation unit 108to select the same unit for processing based on the external informationindicating which unit (frame by frame, field by field) is used in codingthe pictures of the inputted coding video signal 101 and manages thereference pictures stored in a multi frame buffer, that is, the buffer107 frame by frame. More specifically, when the pictures of the inputtedcoded video signal 101 are coded frame by frame in GOP (or sequence) orthe like, the memory management unit 109 controls the motioncompensation unit 108 to perform motion compensation frame by frame, andit performs memory management frame by frame. On the other hand, whenthe pictures of the inputted coded video signal 101 are coded field byfield in GOP (or sequence), the memory management unit 109 controls themotion compensation unit 108 to perform motion compensation field byfield, and it performs memory management frame by frame. Details on thememory management by this video decoder 100 are the same as the one bythe video encoder 70 except that motion compensation is performed usingmotion vectors contained in the inputted coded video signal 101.

Up to this point, two embodiments of a moving picture coding apparatus,a moving picture decoding apparatus and a memory management concerningthe present invention have been explained, and this present invention isnot limited to these embodiments.

There is no need to equip both the storaged flag 44 and the long termflag 45 like this embodiment, in other words, it is possible to use onlyone of them or employ another method.

Further, it is possible to perform memory management frame by frame orfield by field in a mixed manner instead of only frame by frame in afixed manner when considering that improving the function of a memorymanagement and the efficiency in using a buffer frame outweighsimplifying memory management. For example, it is possible to performmanagement on “used”/“unused” frame by frame and on “short term picturememory area”/“long term picture memory area” field by field by assigning1 bit of storaged flag and 2 bits of long term flag to each of thememory areas.

Memory management (“used”/“unused”, “long term picture memoryarea”/“short term picture memory area”) is realized by using flagscorresponding to pictures (frames or fields) in the above-mentionedembodiment, the present invention is not limited to memory managementwhere flags are used. For example, it is also possible to performsimilar memory management by making a management table where picturenumbers of pictures (or reference indexes or numbers showing each areasof a buffer) in a state of “used” (or “unused”) are selected from thepictures stored in the buffer are recorded or a management table wherepicture numbers of pictures (or reference indexes or numbers showingeach areas of a buffer) stored in a “short term picture memory area” (or“long term picture memory area”) are recorded.

Also, it is possible to realize the video encoder and the video decoderthat equip a memory management function shown in the above-mentionedembodiment as a program. Distributing such a program after recording iton a storage medium such as a flexible disc makes it possible to makegeneral-purpose computers in every places function as a video encoder ora video decoder concerning the present invention.

FIG. 13 is an illustration of a storage medium for storing a program forrealizing the video encoder and the video decoder of the above-mentionedembodiment using a computer system. FIG. 13A shows an example of aphysical format of a flexible disc as a recording medium body. FIG. 13Bshows a flexible disc and the front view and the cross-sectional view ofthe appearance of the flexible disc. A flexible disc (FD) is containedin a case F, a plurality of tracks (Tr) are formed concentrically on thesurface of the disc from the periphery into the inner radius of thedisc, and each track is divided into 16 sectors (Se) in the angulardirection. Therefore, in the case of the flexible disc storing theabove-mentioned program, the video encoder and the video decoder as theprogram is recorded in an area allocated for it on the flexible disc(FD). Also, FIG. 13C shows the structure for recording and reading outthe program on the flexible disc (FD). When the program is recorded onthe flexible disc (FD), the computer system (Cs) writes in the videoencoder or the video decoder as a program via a flexible disc drive.When the video encoder and the video decoder mentioned above areconstructed in the computer system by the program on the flexible disc,the program is read out from the flexible disc through a flexible discdrive and transferred to the computer system. Note that the recordingmedium is not limited to flexible discs, and optical discs such asDC-ROMs and DVD-ROMs, memory cards, ROM cassettes and the like can alsobe used.

Also, the video encoder and the video decoder concerning the presentinvention can be applied for various kinds of apparatuses or systems.Applications of the video encoder and the video decoder in theabove-mentioned embodiment will be explained below.

FIG. 14 is a block diagram showing the overall configuration of acontent supply system ex100 for realizing content distribution service.The area for providing communication service is divided into cells ofdesired sizes, and cell sites ex107 to ex110 of fixed wireless stationsare placed in the respective cells. This content supply system ex100 isconnected to each apparatus such as a computer ex111, a Personal DigitalAssistant (PDA) ex112, a camera ex113, a cellular phone ex114 and acellular phone with a camera ex115 via, for example, a combination ofthe Internet ex101, an Internet service provider ex102, a telephonenetwork ex104 and cell sites ex107 to ex110. However, the content supplysystem ex100 is not limited to the configuration as shown in FIG. 14,and may be connected to a combination of any of them. Also, eachapparatus can be connected directly to the telephone network ex104, notthrough the cell sites as fixed radio stations ex107 to ex110.

The camera ex113 is an apparatus capable of shooting video (movingpictures) such as a digital video camera. The cell phone can be a cellphone of a Personal Digital Communications (PDC) system, a Code DivisionMultiple Access (CDMA) system, a Wideband-Code Division Multiple Access(W-CDMA) system or a cellular phone using a Global System for MobileCommunications (GSM) system, a Personal Handy-phone system (PHS) or thelike.

A streaming server ex103 is connected to the camera ex113 via the cellsite ex109 and the telephone network ex104, which enables livedistribution or the like using the camera ex113 based on the coded datatransmitted from the user. Either the camera ex113 or the server fortransmitting the data can code the shot data. Also, the moving picturedata shot by a camera ex116 can be transmitted to the streaming serverex103 via the computer ex111. The camera ex116 is an apparatus capableof shooting still and moving pictures such as a digital camera. In thiscase, either the camera ex116 or the computer ex111 can code the movingpicture data. An LSI ex117 included in the computer ex111 or the cameraex116 performs coding processing. Software for coding and decodingpictures can be integrated into any type of storage media (such as

CD-ROMs, flexible discs, hard discs and the like) that is a recordingmedium which is readable by the computer ex111 or the like. Furthermore,a cellular phone with a camera ex115 can transmit the moving picturedata. This moving picture data is the data coded by the LSI included inthe cellular phone ex115.

The contents supply system ex100 codes contents (such as a music livevideo) shot by users using the camera ex113, the camera ex116 or thelike in the same manner as the above-mentioned embodiments and transmitsthem to the streaming server ex103, while the streaming server ex103makes stream distribution of the contents data to the clients upon theirrequest. The clients include the computer ex111, the PDA ex112, thecamera ex113, the cellular phone ex114 and so on that are capable ofdecoding the above-mentioned coded data. In this way, the contentssupply system ex100 enables the clients to receive and reproduce thecoded data, and further to receive, decode and reproduce the data inreal time so as to realize personal broadcasting.

When each apparatus in this system performs coding or decoding, thevideo encoder or the video decoder can be used, as shown in theabove-mentioned embodiments.

A cellular phone will be explained as an example of the apparatus. FIG.15 is a diagram showing the cellular phone ex115 using the video encoderand the video decoder explained in the above-mentioned embodiments. Thecellular phone ex115 has an antenna ex201 for communicating with thecell site ex110 via radio waves, a camera unit ex203 capable of shootingmoving and still pictures such as a CCD camera, a display unit ex202such as a liquid crystal display for displaying the data obtained bydecoding pictures and the like shot by the camera unit ex203 andreceived by the antenna ex201, a body unit including a set of operationkeys ex204, a voice output unit ex208 such as a speaker for outputtingvoices, a voice input unit 205 such as a microphone for inputtingvoices, a storage medium ex207 for storing coded or decoded data such asdata of moving or still pictures shot by the camera, data of receivede-mail and data of moving or still pictures, and a slot unit ex206operable to attach the storage medium ex207 to the cellular phone ex115.The storage medium ex207 is equipped with a flash memory element, a kindof Electrically Erasable and Programmable Read Only Memory (EEPROM) thatis an electrically erasable and rewritable nonvolatile memory, in aplastic case such as SD cards.

Next, the cellular phone ex115 will be explained with reference to FIG.16. In the cellular phone ex115, a main control unit ex311, which isoperable to perform centralized control on each unit of the body unitincluding the display unit ex202 and operation keys ex204, is connectedto a power supply circuit unit ex310, an operation input control unitex304, a picture coding unit ex312, a camera interface unit ex303, aLiquid Crystal Display (LCD) control unit ex302, a picture decoding unitex309, a demultiplexing unit ex308, a recording and reproducing unitex307, a modem circuit unit ex306 and a voice processing unit ex305 toeach other via a synchronous bus ex313.

When a call-end key or a power key is turned ON by a user's operation,the power supply circuit unit ex310 supplies respective components withpower from a battery pack so as to activate the digital cellular phonewith a camera ex115 for making it into a ready state.

In the cell phone ex115, the voice processing unit ex305 converts thevoice signals received by the voice input unit ex205 in conversationmode into digital voice data under the control of the main control unitex311 including a CPU, a ROM and a RAM, the modem circuit unit ex306operable to perform spread spectrum processing of the digital voicedata, and the communication circuit unit ex301 operable to performdigital-to-analog conversion and frequency transform of the data so asto transmit it via the antenna ex201. Also, in the cellular phone ex115,the communication circuit unit ex301 amplifies the data received by theantenna ex201 in conversation mode and performs frequency transform andanalog-to-digital conversion for the data, the modem circuit unit ex306performs inverse spread spectrum processing of the data, and the voiceprocessing unit ex305 converts it into analog voice data so as to outputit via the voice output unit ex208.

Furthermore, when transmitting e-mail in data communication mode, thetext data of the e-mail inputted by operating the operation keys ex204on the body unit is sent out to the main control unit ex311 via theoperation input control unit ex304. In the main control unit ex311,after the modem circuit unit ex306 performs spread spectrum processingof the text data and the communication circuit unit ex301 performsdigital-to-analog conversion and frequency transform for it, the data istransmitted to the cell site ex110 via the antenna ex201.

When picture data is transmitted in data communication mode, the movingpicture data shot by the camera unit ex203 is supplied to the picturecoding unit ex312 via the camera interface unit ex303. When the picturedata is not transmitted, it is also possible to display the picture datashot by the camera unit ex203 directly on the display unit 202 via thecamera interface unit ex303 and the LCD control unit ex302.

The picture coding unit ex312, which includes the picture encoder asexplained in the present invention, compresses and codes the picturedata supplied from the camera unit ex203 using the coding method usedfor the video encoder as shown in the above-mentioned embodiments so asto transform it into coded picture data, and sends it out to thedemultiplexing unit ex308. At this time, the cellular phone ex115 sendsout the voices received by the voice input unit ex205 during shooting bythe camera unit ex203 to the demultiplexing unit ex308 as digital voicedata via the voice processing unit ex305.

The demultiplexing unit ex308 multiplexes the coded picture datasupplied from the picture coding unit ex312 and the voice data suppliedfrom the voice processing unit ex305 using a predetermined method, themodem circuit unit ex306 performs spread spectrum processing on themultiplexed data obtained as a result of the multiplexing, and thecommunication circuit unit ex301 performs digital-to-analog conversionand frequency transform of the data for transmitting via the antennaex201.

As for receiving data of a moving picture file which is linked to a Webpage or the like in data communication mode, the modem circuit unitex306 performs spread spectrum processing of the data received from thecell site ex110 via the antenna ex201, and sends out the multiplexeddata obtained as a result of the processing to the demultiplexing unitex308.

In order to decode the multiplexed data received via the antenna ex201,the demultiplexing unit ex308 separates the multiplexed data into a bitstream of picture data and a bit stream of voice data, and supplies thecurrent coded picture data to the picture decoding unit ex309 and thecurrent voice data to the voice processing unit ex305 respectively viathe synchronous bus ex313.

Next, the picture decoding unit ex309, which includes the video decoderas explained in the above invention, decodes the bit stream of picturedata using the decoding method corresponding to the coding method asshown in the above-mentioned embodiments to generate reproduced movingpicture data, and supplies this data to the display unit ex202 via theLCD control unit ex302, and thus the moving picture data included in amoving picture file linked to a Web page, for instance, is displayed. Atthe same time, the voice processing unit ex305 converts the voice datainto analog voice data, and supplies this data to the voice output unitex208, and thus voice data included in a moving picture file linked to aWeb page, for instance, is reproduced.

The present invention is not limited to the above-mentioned system, andat least either the video encoder or the video decoder in theabove-mentioned embodiments can be incorporated into a digitalbroadcasting system as shown in FIG. 17. Such ground-based or satellitedigital broadcasting has been in the news lately. More specifically, abit stream of video information is transmitted from a broadcast stationex409 to a communication or a broadcast satellite ex410 via radio waves.Upon receipt of it, the broadcast satellite ex410 transmits radio wavesfor broadcasting, a home-use antenna ex406 with a satellite broadcastreception function receives the radio waves, and a television (receiver)ex401, a set top box (STB) ex407 or the like decodes and reproduce thebit stream. The video decoder as shown in the above-mentionedembodiments can be implemented in the reproduction apparatus ex403 forreading off and decoding the bit stream recorded on a storage mediumex402 that is a recording medium such as a CD and a DVD. In this case,the reproduced video signals are displayed on a monitor ex404. It isalso conceived to implement the video decoder in the set top box ex407connected to a cable ex405 for a cable television or the antenna ex406for satellite and/or ground-based broadcasting so as to reproduce themon a monitor ex408 of the television. The video encoder may beincorporated into the television, in stead of in the set top box. Or, acar ex412 having an antenna ex411 can receive signals from the satelliteex410, the cell site ex107 or the like for reproducing moving pictureson a display apparatus such as a car navigation system ex413.

Furthermore, the video encoder shown in the above-mentioned embodimentscan code picture signals for recording on a recording medium. As aconcrete example, there is a recorder ex420 such as a DVD recorder forrecording picture signals on a DVD disc ex421 and a disc recorder forrecording them on a hard disc. They can be recorded on an SD card ex422.If the recorder ex420 includes the video decoder shown in theabove-mentioned embodiment, the picture signals recorded on the DVD discex421 or the SD card ex422 can be reproduced for display on the monitorex408.

Note that a conceivable configuration of the car navigation system ex413is the configuration obtained by eliminating the camera unit ex203, thecamera interface unit ex303 and the picture coding unit ex312 fromexisting components in FIG. 16. The same goes for the computer ex111,the television (receiver) ex401 and the like.

In addition, three types of implementations can be conceived for aterminal such as the above-mentioned cell phone ex114, asending/receiving terminal implemented with both an encoder and adecoder, a sending terminal implemented with an encoder only, and areceiving terminal implemented with a decoder only.

As described above, it is possible to use the video encoder or the videodecoder in the above-mentioned embodiments in any of the above-mentionedapparatuses and systems, and by using this method, the effects explainedin the above embodiments can be obtained.

From the invention thus described, it will be obvious that theembodiments of the invention may be varied in many ways. Such variationsare not to be regarded as a departure from the spirit and scope of theinvention, and all such modifications as would be obvious to one skilledin the art are intended for inclusion within the scope of the followingclaims.

INDUSTRIAL APPLICABILITY

The moving picture coding apparatus, a moving picture decoding apparatusand a memory management device concerning the present invention can beused as a video encoder and a video decoder that are realized bysoftware executed using a electric circuit such as an LSI or a computer.For example, they are useful as a video encoder and a video decoderequipped with computers, PDAs, digital broadcast sending devices,cellular phones and the like which are used for coding moving pictures,decoding and reproducing the coded moving pictures.

1. A picture decoding apparatus that decodes a sequence of codedpictures in a moving picture, the picture decoding apparatus comprising:a memory having a plurality of memory areas in each of which picturedata corresponding to one frame, one field, or two fields composed of atop field and a bottom field can be stored; a decoding unit configuredto: (i) decode the sequence of coded pictures in the moving picture onlyon a frame basis, with reference to frame data stored in the memory; or(ii) decode the sequence of coded pictures in the moving picture only ona field basis, with reference to field data stored in the memory; or(iii) decode the sequence of coded pictures in the moving picture on aframe basis and a field basis, with reference to either the frame dataor the field data stored in the memory according to the coded picturesin the moving picture decoded either on the frame basis or on the fieldbasis; a memory management unit configured to manage a state of eachmemory area only on a frame basis, regardless of whether the sequence ofcoded pictures in the moving picture is decoded only on a frame basis,only on a field basis, or on a frame basis and a field basis; and astorage unit configured to store the decoded frame data and the decodedfield data decoded by the decoding unit into the memory under managementof the memory management unit, wherein, when first field data isreference picture data and is stored in one memory area included in theplurality of memory areas, said memory management unit is configured to:(i) allow second field data to be stored in the one memory area intowhich the first field data is stored when the first field data and thesecond field data constitute a same frame and the second field data isreference picture data; and (ii) prohibit the second field data frombeing stored in the one memory area into which the first field data isstored and allow the second field data to be stored in another memoryarea other than the one memory area and included in the plurality ofmemory areas when the first field data and the second field data do notconstitute a same frame and the second field data is reference picturedata.